You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Render daemonsets.podSecurityContext on GPUCluster DaemonSets - #3222
GPUCluster accepts spec.daemonsets.podSecurityContext, but the DRA driver, DCGM, DCGM exporter, and validator DaemonSets never copied it onto the pod. ClusterPolicy already applies that field through applyCommonDaemonsetConfig.
These four DaemonSets now render the context the same way the driver operand does. Container securityContext is unchanged, including privileged: true and explicit runAsUser values.
Checklist
No secrets, sensitive information, or unrelated changes
Lint checks passing (make lint)
Generated assets in-sync (make validate-generated-assets); no API or generated-asset changes
Go mod artifacts in-sync (make validate-modules); no dependency changes
Test cases are added for new code paths
Testing
go build ./...
go test ./internal/state/ ./internal/render/
go test ./controllers/ -run 'TestClusterPolicyDaemonsetsPodSecurityContext|TestApplyCommonDaemonsetConfig|TestTransformDCGM$|TestTransformDCGMExporter|TestTransformValidator$'
go vet ./internal/state/ ./controllers/ ./internal/render/
gofmt -l on the changed Go files is clean
TestGPUClusterDaemonsetsPodSecurityContext renders each GPUCluster DaemonSet with and without the context. TestClusterPolicyDaemonsetsPodSecurityContext is the control on the ClusterPolicy DCGM, DCGM exporter, and validator assets. Not applied to a live GPU cluster.
GPUCluster accepts spec.daemonsets.podSecurityContext but does not copy it
onto the DRA driver, DCGM, DCGM exporter, or validator pod specs. Render the
context the same way the driver operand does, and leave container privileged
and runAsUser settings in place.
FixesNVIDIA#3203
Signed-off-by: Dundy Pasupuleti <dundysm@gmail.com>
@rahulait ready for review whenever you get a chance; dra driver, dcgm, dcgm exporter and validator now pick up daemonsets.podSecurityContext, with clusterpolicy render tests as the control.
The four GPUCluster DaemonSet templates now conditionally render spec.daemonsets.podSecurityContext as the pod-level security context. Rendering tests check configured and unset values and verify container security contexts. Controller tests also check pod security context handling through preProcessDaemonSet, including behavior when the policy is unset.
Merge Risk: Moderate · up to 76ae4
A configured non-root UID can prevent the DRA GPU plugin from starting on affected nodes. Resolve or explicitly accept that startup risk before merging.
Security architecture risk: Low · up to 76ae4
The change honors existing configuration without adding a new actor or removing container-level security settings. Its scope includes privileged GPU components, so previously configured pod defaults may affect their execution during rollout. Deployment permissions and live admission behavior remain unverified.
Retained concerns
No architecture-level concerns identified.
Security review details
Security Blast Radius
inferred — The configurable scope is the common GPUCluster operand set, not an individual tenant pod. A subject permitted to modify GPUCluster can now affect pod security defaults across these four DaemonSets and their selected nodes. The API declares GPUCluster cluster-scoped and singleton-named; generated enforcement and deployed writer permissions were not verified.
Trust Boundaries and Controls
observed — The runtime configuration path remains the GPUCluster resource read by the reconciler. The routed Test functions and security-context helpers reside in test files, not newly exposed production entrypoints. The supplied value uses the existing Kubernetes PodSecurityContext type.
Resilience and Maintainability Implications
observed — The existing application path attaches controller ownership before creating or updating resources. Failed state synchronization prevents aggregate readiness and causes reconciliation to retry, rather than treating a partially applied operand set as ready.
Hardening Proposals
proposed — Before upgrading an installation with an existing pod security context, validate its compatibility with all four operands, especially runAsNonRoot settings alongside required UID 0 containers. Exercise configuration removal during rollout to confirm the deployed environment converges as expected.
- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 1
Additional comments: 1
internal/state/gpucluster_render_test.go: 1
172-184: Trivial · Maintainability & Code Quality · Low value
Unverified finding
Verification ran but could not confirm this finding. It is shown for review, not as a verified issue.
The required map omits the dra-validator state.
The dra-validator state is rendered but has no container expectations. For it, the test checks only the pod-level context and that container contexts are unchanged. That is acceptable because the validator template sets no container securityContext. A later change that adds one would not be caught.
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.
containerd privileged non root exec ambient capabilities CAP_SYS_ADMIN bash kubernetes runc UID 1000 privileged mount
Result:
Sources consulted:
- Chapter 7: Security Boundaries - Containerd From The Bottom Up — https://thecontainerdbook.com/chapters/part-2/07-security
- Kubernetes Linux Capabilities [Add, Drop and Examples] — https://www.golinuxcloud.com/kubernetes-securitycontext-capabilities/
- Docker and Container Capabilities | Luke Hebert — https://lukehebert.com/post/docker-and-container-capabilities/
- libcontainer/SPEC.md at main · opencontainers/runc — https://github.com/opencontainers/runc/blob/main/libcontainer/SPEC.md
- Capabilities and Privilege Management | open-beagle/runc | DeepWiki — https://deepwiki.com/open-beagle/runc/6.3-capabilities-and-privilege-management
Sources consulted:
- site/content/docs/reference/helm-values.md — https://github.com/kubernetes-sigs/dra-driver-nvidia-gpu/blob/main/site/content/docs/reference/helm-values.md
- deployments/helm/dra-driver-nvidia-gpu/values.yaml — https://github.com/kubernetes-sigs/dra-driver-nvidia-gpu/blob/main/deployments/helm/dra-driver-nvidia-gpu/values.yaml
- deployments/helm/dra-driver-nvidia-gpu/values.yaml — https://github.com/kubernetes-sigs/dra-driver-nvidia-gpu/blob/v0.4.1/deployments/helm/dra-driver-nvidia-gpu/values.yaml
- 616fe03 Add SCC to service accounts on OpenShift — https://github.com/kubernetes-sigs/dra-driver-nvidia-gpu/commit/616fe03385145ca5dfeb8db5482d32e19b0e0b4e
- Add MPS config option to configure nvidia-cuda-mps-control to run in multi user mode — GitHub pull request 1153 in kubernetes-sigs/dra-driver-nvidia-gpu (link omitted to avoid creating a cross-reference)
containerd CRI privileged runAsUser non root ambient capabilities 2.3 exec all capabilities Kubernetes
Result:
Sources consulted:
- Kubernetes should configure the ambient capability set — GitHub issue 56374 in kubernetes/kubernetes (link omitted to avoid creating a cross-reference)
- How to Restrict Container Capabilities Using securityContext in Kubernetes — https://oneuptime.com/blog/post/2026-02-09-restrict-container-capabilities-security-context/view
- Chapter 7: Security Boundaries - Containerd From The Bottom Up — https://thecontainerdbook.com/chapters/part-2/07-security
- Capabilities - Building a Container from Scratch in Rust — https://brianshih1.github.io/mini-container/capabilities.html
- containerd should set ambient capabilities — GitHub issue 5644 in containerd/containerd (link omitted to avoid creating a cross-reference)
Script executed:
python3 - <<'PY'import urllib.requesturls = ['https://raw.githubusercontent.com/containerd/containerd/v2.3.5/internal/cri/server/container_create.go','https://raw.githubusercontent.com/kubernetes-sigs/dra-driver-nvidia-gpu/v0.5.1/cmd/gpu-kubelet-plugin/driver.go','https://raw.githubusercontent.com/kubernetes-sigs/dra-driver-nvidia-gpu/v0.5.1/deployments/helm/dra-driver-nvidia-gpu/templates/kubeletplugin.yaml',]for url in urls: print('\n###', url) try: text = urllib.request.urlopen(url, timeout=12).read().decode() lines = text.splitlines() needles = ('WithoutAmbientCaps', 'WithAllCurrentCapabilities', 'Privileged', 'kubeletplugin.Start(', 'Mkdir', 'runAsUser', 'podSecurityContext') hits = [i for i, line in enumerate(lines) if any(n in line for n in needles)] for i in hits: print('\n'.join(f'{j+1}: {lines[j]}' for j in range(max(0,i-4), min(len(lines),i+7)))) print('---') if not hits: print('No selected symbols found') except Exception as exc: print(type(exc).__name__, str(exc))PY
Repository: NVIDIA/gpu-operator
Length of output: 8306
Script executed:
python3 - <<'PY'import json, urllib.requestbase='https://raw.githubusercontent.com/kubernetes-sigs/dra-driver-nvidia-gpu/v0.5.1/'def read(url): return urllib.request.urlopen(url, timeout=15).read().decode()try: tree=json.loads(read('https://api.github.com/repos/kubernetes-sigs/dra-driver-nvidia-gpu/git/trees/v0.5.1?recursive=1'))['tree'] paths=[x['path'] for x in tree if x['type']=='blob' and ((x['path'].startswith('cmd/gpu-kubelet-plugin/') and x['path'].endswith('.go')) or x['path'].endswith('Dockerfile') or x['path']=='go.mod')] print('Candidate files:', ', '.join(paths)) for path in paths: if path.endswith('Dockerfile') and 'gpu' not in path and path != 'Dockerfile': continue try: lines=read(base+path).splitlines() except Exception as e: print(path, 'read failed:', e) continue needles=('func NewDeviceState(', 'os.Mkdir', 'MkdirAll(', 'k8s.io/dynamic-resource-allocation', 'USER ', 'FROM ', 'kubeletplugin.Start(', 'NewDriver(ctx') hits=[i for i,l in enumerate(lines) if any(n in l for n in needles)] if hits: print('\n###', base+path) for i in hits: print('\n'.join(f'{j+1}: {lines[j]}' for j in range(max(0,i-5),min(len(lines),i+12)))) print('---')except Exception as e: print('Inspection failed:', type(e).__name__, e)PY
Repository: NVIDIA/gpu-operator
Length of output: 30528
Script executed:
python3 - <<'PY'import urllib.requestbase='https://raw.githubusercontent.com/kubernetes-sigs/dra-driver-nvidia-gpu/v0.5.1/'for path, ranges in {'cmd/gpu-kubelet-plugin/main.go':[(44,80),(110,150),(280,370),(375,430)],'deployments/container/Dockerfile':[(1,200)],}.items(): print('\n###',base+path) try: lines=urllib.request.urlopen(base+path,timeout=15).read().decode().splitlines() for a,b in ranges: print('\n'.join(f'{i+1}: {lines[i]}' for i in range(a-1,min(b,len(lines))))) except Exception as e: print(type(e).__name__,e)PY
cat -n tests/holodeck.yaml | grep -n -C 2 'containerd'
cat -n manifests/state-dra-driver/0500_daemonset.yaml | sed -n '192,205p;227,245p;363,370p'
Repository: NVIDIA/gpu-operator
Length of output: 18786
Script executed:
python3 - <<'PY'import urllib.requesturl='https://raw.githubusercontent.com/kubernetes-sigs/dra-driver-nvidia-gpu/v0.5.1/cmd/gpu-kubelet-plugin/main.go'lines=urllib.request.urlopen(url,timeout=15).read().decode().splitlines()for a,b in [(78,88),(190,220),(245,280)]: print(f'\n### {url}:{a}-{b}') print('\n'.join(f'{i+1}: {lines[i]}' for i in range(a-1,min(b,len(lines)))))for i,l in enumerate(lines): if 'func main(' in l or 'RunPlugin(' in l or 'app.Run' in l: print(f'\n### {url}:{max(1,i-5)}-{min(len(lines),i+12)}') print('\n'.join(f'{j+1}: {lines[j]}' for j in range(max(0,i-6),min(len(lines),i+12))))PY
Sources consulted:
- capabilities(7) - Linux manual page — https://man7.org/linux/man-pages/man7/capabilities.7.html
- Linux capabilities: what the five sets actually do — RodHat — https://rodhat.com/tips/2026-09-20-linux-capabilities-set-model/
- How to execve a process, retaining capabilities in spite of missing filesystem-based capabilities? — https://stackoverflow.com/questions/14636359/how-to-execve-a-process-retaining-capabilities-in-spite-of-missing-filesystem-b
- Beyond Container Capabilities: Understanding Linux Capability Sets | utam0k — https://www.utam0k.jp/en/blog/2025/12/14/linux-capability-sets/
- capabilities: Ambient capabilities [LWN.net] — https://lwn.net/Articles/636533/
Script executed:
python3 - <<'PY'from urllib.request import urlopenurl='https://raw.githubusercontent.com/kubernetes-sigs/dra-driver-nvidia-gpu/v0.5.1/cmd/gpu-kubelet-plugin/main.go'lines=urlopen(url,timeout=15).read().decode().splitlines()for i,line in enumerate(lines): if 'KUBELET_PLUGINS_DIRECTORY_PATH' in line: print(url) print('\n'.join(f'{j+1}: {lines[j]}' for j in range(max(0,i-5),min(len(lines),i+3))))PY
Repository: NVIDIA/gpu-operator
Length of output: 642
Ensure the GPU plugin can start with a non-root pod UID.
With runAsUser: 1000 on containerd 2.3.5, the plugin can fail to create its directory under a root-owned kubelet plugins directory. The gpus container then exits and can crash-loop. Privileged mode does not preserve ambient capabilities across the non-root executable.
The render test does not detect this startup failure.
The reason will be displayed to describe this comment to others. Learn more.
this only changes anything when daemonsets.podSecurityContext is set, and that matches clusterpolicy today: applyCommonDaemonsetConfig copies the same field onto every operand daemonset, including the device plugin that also writes under /var/lib/kubelet. the init container here already pins runAsUser: 0, and i left the plugin containers as they are so the change stays a straight port of the clusterpolicy behavior. @rahulait if you'd prefer the kubelet plugin containers to pin runAsUser: 0 as well so a non-root pod uid can't reach them, i can add that here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue
Fixes #3203
Description
GPUCluster accepts
spec.daemonsets.podSecurityContext, but the DRA driver, DCGM, DCGM exporter, and validator DaemonSets never copied it onto the pod. ClusterPolicy already applies that field throughapplyCommonDaemonsetConfig.These four DaemonSets now render the context the same way the driver operand does. Container
securityContextis unchanged, includingprivileged: trueand explicitrunAsUservalues.Checklist
make lint)make validate-generated-assets); no API or generated-asset changesmake validate-modules); no dependency changesTesting
go build ./...go test ./internal/state/ ./internal/render/go test ./controllers/ -run 'TestClusterPolicyDaemonsetsPodSecurityContext|TestApplyCommonDaemonsetConfig|TestTransformDCGM$|TestTransformDCGMExporter|TestTransformValidator$'go vet ./internal/state/ ./controllers/ ./internal/render/gofmt -lon the changed Go files is cleanTestGPUClusterDaemonsetsPodSecurityContextrenders each GPUCluster DaemonSet with and without the context.TestClusterPolicyDaemonsetsPodSecurityContextis the control on the ClusterPolicy DCGM, DCGM exporter, and validator assets. Not applied to a live GPU cluster.